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Technical  Information 


1.  Technical  Progress  /  Highlights  -  Observations 

The  early  results  of  the  Algebra  Challenge  (detailed  in  the  previous  report)  showed  that  mastery  of  linear 
equation  solving  can  be  reached  in  just  1.5  hours  of  play.  More  importantly,  that  mastery  is  not  reached 
by  a  select  few,  but  by  the  nearly  the  whole  classroom.  The  average  mastery  rate  for  all  grades  was  92.9% 
after  1.5  hours.  This  is  especially  surprising  since  many  of  the  participants  were  in  elementary  school, 
where  algebra  is  usually  not  considered  (it  is  commonly  a  milestone  for  7th  and  8th  graders).  We  believe 
the  adaptive  nature  of  the  game  was  instrumental  in  achieving  these  results.  Only  a  tiny  percentage  of 
students  finished  the  game  without  requiring  at  least  some  remediation.  Some  grades  required 
significantly  more  effort  and  review  than  others.  The  game  was  able  to  adapt  in  real-time  to  each  student 
in  order  to  maximize  the  likelihood  of  mastery.  As  a  result,  some  students  played  almost  10  times  more 
levels  than  others.  The  engagement  structures  of  the  game  ensured  that  they  persisted  through  this 
challenge. 

We  continued  to  improve  our  games  and  game -based  learning  tools  over  the  summer  quarter.  We 
implemented  a  revamped  interface  for  the  Teacher  portal  to  improve  ease  of  use  for  teachers  to  enter  and 
manage  on  student  accounts.  We  also  worked  with  a  group  of  math  teachers  to  create  specific  lesson  units 
that  use  the  games  as  a  teaching  tool  and  which  can  be  integrated  into  a  standard  math  curriculum.  Our 
plan  is  to  begin  trials  with  those  units  next  year.  All  of  the  games  are  now  running  on  iPad  and  Android 
platforms  and  suitable  for  play  on  tablets,  so  some  of  the  trials  may  take  place  using  the  teacher  tablet 
system  that  is  currently  being  co-developed  with  Engaged  Learning. 

Based  on  item  analysis  results  we  decided  to  remove  what  we  had  previously  been  calling  “far  transfer” 
questions  from  each  game  assessment,  and  to  focus  instead  on  only  testing  on  material  most  relevant  to 
the  game  in  which  the  assessment  is  embedded. 

The  “Algebra  Adventure”  or  word  problems  game  is  still  undergoing  development.  In  this  game,  students 
literally  "pull"  values  and  concepts  from  the  story  and  form  them  into  an  equation.  The  equations  are  then 
combined  with  each  other  to  determine  the  final  answer.  The  largest  challenge  here  at  the  moment  is 
designing  a  user  interface  that  is  meaningful  and  intuitive.  We  expect  to  begin  playtesting  the  interface  in 
the  next  reporting  period  as  well  as  create  a  robust  system  for  generating  these  word  problems  in  specific 
progressions  and  with  specific  “themes”  (such  as  word  problems  that  take  place  in  a  fantasy  world, 
science  fiction  world  etc.) 

We  are  in  the  planning  stages  of  more  large-scale  and  statewide  challenges,  with  one  being  developed  in 
Norway  for  late  this  year  and  one  in  Minnesota  for  early  next  year.  Additional  challenges  will  help  us 
maximize  the  percentage  of  mastery  for  a  greater  set  of  common  core  concepts. 

2.  Results  or  Problems  and  Solutions 

Due  to  the  large  volume  of  data  we  have  collected  we  are  looking  to  hire  an  additional  grad  student  or 
postgrad  to  help  with  data  analysis.  One  of  these  datasets  is  the  Baker  data  set,  in  which  several 
researchers  observed  students  playing  Treefrog  Treasure,  Creature  Capture,  and  Refraction.  They  coded 
the  students  for  affective  and  behavioral  states  while  playing;  these  were  synced  up  with  the  replay  data. 
The  hope  is  that  we  can  learn  classifiers  which  can  tell  if  students  are  confused,  frustrated,  bored,  happy, 
concentrating,  etc.  while  playing  based  solely  on  game  play  data. 

We  also  have  many  datasets  collected  from  various  game  sites.  One  in  particular  is  a  Treefrog  Treasure 
dataset  in  which  numberlines  are  given  out  nearly  at  random,  with  the  denominator,  numerator, 
representation,  hinting  system,  and  fraction  animations  all  changed  at  every  numberline.  The  idea  here  is 
to  learn  from  data  what  pattern  of  numberlines  contributes  to  the  greatest  engagement  and  ability  to  solve 


future  numberlines.  How  players  interact  with  our  games  may  give  us  some  information  about  what  they 
will  do  in  the  future,  how  well  they  understand  the  content,  or  what  emotions  they  are  currently 
experiencing. 

In  most  cases,  the  most  difficult  part  of  these  tasks  is  not  the  algorithm  used  to  issue  the  predictions,  but 
the  construction  of  features,  that  is:  what  parts  of  the  gameplay  should  we  pay  attention  to?  We  could  for 
example  evaluate  the  time  it  takes  to  beat  levels,  the  amount  that  players  re -try  the  same  strategy  over  and 
over,  or  how  shallow  their  search  strategy  appears  to  be.  Therefore  one  of  our  projects  will  be  to 
automatically  generate  many  of  these  features  by  taking  advantage  of  the  fact  that  players  are  in  a  game 
with  a  clearly  defined  goal.  We  will  do  this  by  creating  a  state  space  of  different  game  states  and 
constructing  those  features  on  the  resulting  search  graph.  We  will  be  developing  this  using  data  gathered 
from  both  Refraction  and  Treefrog  Treasure,  with  the  intent  of  predicting  affective  states  of  players  and 
the  ability  to  answer  numberlines  correctly.  That  in  turn  will  feed  back  into  our  automated  assessment  and 
adaptivity. 

3.  Significant  Accomplishments  Anticipated  During  Next  Reporting  Period 

•  Perform  trials  of  new  game-based  curriculum 

•  New  format  for  encoding  game  progressions  so  that  they  can  be  “called”  by  an  external  app 

•  Improvements  to  enable  real-time  feedback  for  teachers 

4.  Publications  relevant  to  this  effort 

"A  Mixed-Initiative  Tool  for  Designing  Level  Progressions  in  Games”  -  Eric  Butler,  Adam  M.  Smith, 
Yun-En  Liu,  and  Zoran  Popovic.  ACM  Symposium  on  User  Interface  Software  and  Technology  (UIST) 
2013 

5.  Meetings  and  Events  (Please  include  meetings  with  subcontractors) 

August  19-22  -  Serious  Play  Conference.  Redmond,  WA.  “Evidence  for  Game-Based  Development  of 
World-Class  Expertise  and  School  Mastery” 

6.  Changes  to  the  Contract  Organization 


None. 


